Lyrics Report

This first report of the lyrics dataset will be an exploration of the dataset and it will contain:

1. Input data description and overview
2. Data processing steps and methods
3. Data exploration
4. Next steps

After understanding the data and preprocessing it to contain useful features, I will use analytical approach to find some dependencies or correlations to be able create a meaningful model based on my findings in the next report.

Input data description and overview¶

The lyrics dataset contains popular songs with lyrics. There are six columns: id, song name, year of publishing, interpret name, assigned genre and the lyrics. Some rows are music composition without lyrics. There are hyphens instead of spaces in names of songs and interprets. The only numeric column is year, while the others are strings and will be treated as categorical variables.

In this section, we will present a snapshot of the dataset and delve into its fundamental characteristics before proceeding with further processing.

The dataset contains 362237 songs, but only 266557 of them contain any lyrics. This means that it is quite big data (it is one of the largest datasets), especially after featurization of the text. Therefore, I will be doing basic analysis using all samples, but some analyses and modelling will be done only using the songs containing lyrics.

Next, you can see a few charts showing the distribution of values in individual columns. For better readability, I have replaced "-" in song and interpreter names with spaces and capitalized first letters.

We can see the distribution of individual years in the first chart. We can see that most songs were published in 2006 or 2007, so we should be aware that the results reflect rather just the specific culture of those times.

The table also suggests that there might be some data quality issues regarding the release year since there are 10 songs with the year 2038 and are also a few from a distant past.

In the second chart, we can see the distribution of genres. The dataset is quite biased with its genre representation, so the results cannot be easily generalized to all music. We also have 29814 songs where the genre is not available and 23683 songs where the genre is labeled as "Other".

The last chart from this section shows the number of songs of the top 20 interpreters based on their song count. There are 18231 unique interpreters in the dataset, so we coudn't show all of them.

Although there are many interpreters and there are many that have a lot of songs, there are also 4514 that only have one song and 1388 of them only have two songs as we can see in the following table.

No. of songs	No. of interpreters
1	4514
2	1388
3	832
4	600
5	490
6	450
7	421
8	475
9	503
10	763

Data processing steps and methods¶

We have already made the first processing step of replacing hyphens with spaces and capitalizing first results. Now we need to take care of data quality issues and then featurize the song lyrics.

1. Data quality¶

Some of the most frequent data quality issues are null values, duplicates, inconsistencies or outliers. I have found no duplicate rows in the dataset. But we have discovered a few data quality issues in the first part - missing lyrics and release years in distant past or in the future. So first, we will have a look at columns with null values.

There are 95680 songs with missing lyrics. For the first analysis, we will just impute empty strings "" in those fields.

There are also 2 songs with missing song_name. We can see these rows in the table below.

These songs seem to be mistakes, so we can drop them.

We can notice from this table that some interpreters can sometimes collaborate with others so we need to be careful that they might also be present in other interpreter names.

Next, we will have a look at rows with suspicious release years.

These rows also seem to be errors. Since we have enough samples, we can also exclude these rows from the dataset.

This table revealed another data quality issue - in song names, abbreviations like let's or can't are written the same way as if there were spaces instead of apostrophes. However, this should not be much of a problem since it is consistent between interpreters and song name so we will leave it as is.

Looking at the data also reveals that not all songs are in English. For example the following interpreter sings in German. We will need to take this into consideration when creating a model since creating a single model determining interpreter or genre out of lyrics will be greatly influenced by the language of the song. However, we will not adjust the data for this yet.

There are also inconsistencies in song lyrics and the style in which they are written. Some contain spelling mistakes, but there is not much we can do about that. We could see in the above tables that some lyrics contain parts like "Verse 1:" or "[Chorus]". This would be useful if it was everywhere, but since it is not, it would be better to remove them. We will remove all parts in [] brackets and add "Chorus" and "Verse" to stopwords used when making a model.

Deeper investigation reveals that there are also phrases like "(instrumental)", "(guitar, vocals)", "(x6)", etc. Although things in regular brackets can be relevant and mostly they are some background chorus, there is a lot of noise inside them and their appearance depends on the style of the person who wrote the lyrics rather than on the actual lyrics, so we will remove them as well.

Other data quality issues revealed include for example having the song names instead of lyrics or other non-lyrics parts such as names of people who sing the part.

Because of complexity, we will not address these issues now, however it would be beneficial to address them for business usage of the dataset.

Also, although there are no duplicated rows, some songs can be present multiple times in different versions. There are many song names that appear multiple times, but only some of them are remakes. Some authors can also be present under multiple names.

See some examples of these issues below.

Since the ultimate goal is to be able to classify interpreters, it will be best to have less interpreter names, so I decided to try to find similar interpreter names and replace them with the same value. As the first step, I went through all interpreter names and found those that have the same words in the name, just in a different order, and unified them.

Then there are also many songs that are present multiple times - sometimes as remakes, sometimes instrumental versions etc. Since we have a very large dataset, it would be better to get rid of them. It also gives us a chance to reveal more interpreters that are present multiple times under different names. So I found all songs with the same name and genre and decided which of them are by the same interpreter based on fuzzywuzzy similarity score of the interpreters. Then I unified interpreter names across the dataset and kept only the version with the lyrics, and when there were more of them, only the oldest one. If it still was a tie, I chose the one with longer lyrics.

This method does many mistakes (it can change the name of one interpreter to a different one) and if we were to productionalize the dataset, it would need to be taken care of more carefully. But based on our purposes I think we will gain more value from having fewer interpreters than we would have from being 100 % accurate.

2. Featurization¶

Since words cannot be used for modelling as they are, we will need to create some features to be able to analyze them further. We will come back again to featurization in modelling stage since right now I cannot be sure which features we will need. But for exploration, I will create the following features:

• Word count: the total number of words in the lyrics
• Unique word count: the number of unique words in the lyrics
• Average word length: the average length of the words in the lyrics
• Word frequencies (TF-IDF featurization will be used in modelling, however, it is not so good for exploration purposes)

Data exploration¶

1. Word count and length¶

Now we will examine our created features. The following table shows some basic statistic values of the distribution of word count, unique word count and average word length.

On the following chart, you can see the distribution of word count in the dataset. There are many songs that have no words as already stated, but most songs have around 200 words. However, there are also many outliers with over a thousand words that were hidden in the chart.

The following boxplot shows the distribution of wordcount and unique word count in different genres. Again, some outliers have been hidden since they would make the chart difficult to read. The red triange is the mean value.

This plot seems to contain a lot of useful information. For example, we can see that there are genres that are very rarely without lyrics as well as genres that very often have no lyrics. Above all, songs labeled as "Other" rarely have any lyrics. Jazz and Electronic have at least half of their songs with no lyrics. Hip-Hop also has many songs with no lyrics, however, on average, it has the highest amount of words as well as unique words. Although there are many words in pop, there are not many unique ones.

Not Available will probably not be a very useful category since it just seems to be an average case.

The following chart shows the same information per interpreter. It shows information for 50 interpreters with the highest number of songs.

Next, we can see the length of words in the lyrics. This information can be quite biased by lyrics quality, but it could still give us some useful information.

Word length does not differ very much between genres and interpreters. Generally Metal has the longest words and also the largest variance. One explanation that comes to my mind is that it might often be in a different language, but we can come back to this hypothesis in modelling stage. The word length is mostly around 4.

There are a few interpreters whose word length is different from the mean. However, my investigation revealed that it is really most likely because of the style in which the lyrics are written. We can sometiomes see words in the style of "oh-oh-oh" and "oh oh oh" in other cases. That means that this feature might be sometimes a good predictor in our dataset, but it would not be a good predictor for generalizing on data from new sources.

2. Word frequencies¶

The following chart shows the frequency of individual words in the lyrics, after some basic stopwords were removed. The smaller stopwords version was used with some modifications, since in songs there can be quite a lot of meaning in otherwise meaningless words.

<Axes: title={'center': 'Word frequencies in different genres'}, xlabel='Word', ylabel='No. of occurences'>

We can see that term frequency does not differ very much between genres. The chart also contains some spanish words like "que", "de" and "la". We can see that songs labelled as "Other" or "Not Available" might often be in Spanish based on their distribution of the word "que", as opposed to Rock which contains fewer occurences of this word. The main genre with a different word distribution is Hip-Hop with its few occurences of "love" or "away", "heart" and many occurences of "like" or "get"/"got". Otherwise the distributions are quite similar.

3. Multigenre interpreters¶

There are 11 interpreters in the dataset that sing in two genres (after excluding Not Available). You can see them in the following table. Since it is so rare, it means that genre will be a good predictor for interpreter.

Conclusion¶

We have seen some basic features and statistics of our dataset. We also cleaned our data and prepared some useful features. We have gained good understanding of our data and now we can use them in further analyses and modelling.

Based on the results, in my next report, I would like to dedicate space to the following topics:

• separating songs based on their language (either by using a library or by unsupervised clustering): this could significantly improve performance of any other model
• prediction of song genre/interpreter/year according to lyrics